texts: Defining State Representation in Deep Q-Learning

texts:
So I am having difficulty difficulty figuring out exactly how I want to represent my environment state in my Deep Q-learning problem.

Premise: There is a 2D grid space of which an agent needs to visit all the cells (because it is searching the space), once a cell is visited it is considered "searched" and therefore does not need to be visited again.

First Take: I thought well what if the state representation of the environment to the agent was just the 8 adjacent cells and their searched/unsearched status. In this case the resulting actions would be to move to one of the adjecent cells. Here we have 2^8 different possible states and then 8 actions for our Q(s,a) function.

Since I am using a DQN structure to the problem, the Q function would be approximated by a neural network with 8 inputs (status of adjacent cells) and 8 outputs (actions). Obviously the reward function would be simple as it yields a high reward for moving to a cell that has not been searched and a low reward otherwise.

However, this presents an obvious problem when all the adjecent cells have been searched. The agent would move randomly and possibly away from cells that need to be searched. This is not good for my purposes.

Second Take: I thought if I wanted to work around this I would need to incorporate information about the entire state-space into my Q function. Meaning that the agent knows the status of all cells and has actions that are basically "waypoints" that bring the agent to that cell.

This would make my inputs and outputs to the Q function A LOT larger as I have a really big state-space. I don't have a specific size yet, but it could easily be 10,000 (100x100 grid) or larger. And obviously my reward function would need to reward the agent more for actions that involve less movement.

Also, my problem can be formulated in a simulation to train the Q function, so I'm not worried about training time due to physical constraints.

So basically my question is this:

Can this Q function still be approximated by a neural network of reasonable size? Yes, I have a lot of processing power at my disposal (sorry I know this is all very vague), but I'd rather not blow things out of proportion.

tldr: I have Deep Q-Learning problem where I think I need a large state-action space, and am wondering if it is too large to be reasonably approximated by a neural network.

Edit: As @Constantinos noted, I didn't really define my problem well enough, so here are more assumptions and definitions below.

State-Space:

The grid itself is not exactly the state-space, each cell in the grid has a "searched" or "unsearched" status so the total number of unique states that my state-space can take on is really along the lines of 2^10,000 (assuming 100x100 grid).
Additionally, the state-space also contains information about the location of the agent.

Action-Space:

At any given position, the agent may move to any other location on the grid. Therefor the action space is simply the 10,000 moves available.

Reward Function:

Positive reward for choosing action that brings agent to unsearched location.
Negative reward for choosing action that brings agent to location that is far away (to constrain movement).
High positive reward for choosing action that brings the agent to a goal state (i.e. where all locations have been searched).

I/O of Q-Network:

The input will be the a vector of elements (each representing a cell on the grid, can also be seen as an 'image feature' where the image is the grid) that take on values 0 or 1 (unsearched or searched).

The input will also have to include some embedded information on the agents location (I am currently thinking of just altering the value of the input element that represents the agents location cell to be -1).
Output of the network will be a vector of action utility values, each representing the utility of moving to a certain grid location.

Lastly, I should note that once an agent visits a location, that location is now considered "searched" and the state of the environment is updated accordingly.


texts: Can Hadoop be beneficial when data is in database tables and not in a file system

texts: I work for a bank. Most of our data is in the form of database tables. Would we benefit by implementing Hadoop? I am of the impression that Hadoop is more for a Distributed File System (unstructured data) as opposed to OLAP databases (Netezza)


texts: More pre-trained embeddings for PyTorch Big Graph

texts: Apart from the entire wikidata, are there any other PyTorch Big Graph pre-trained graph embeddings on smaller sized knowledge graph, like freebase-15k? I do not have the resources to build it from scratch.


texts:
I have two csv files:

sub_compiler.to_csv('sub_compiler.csv')
sub_compiler.head()

and

sub_opt = pd.read_csv('sub_opt.csv')
sub_opt.head()

and what I would like to do is to create a csv file where I have something of the form

compiler, opt

How could I do this? I need to do this to make a submission.

Thank's in advance.

[EDIT] Thank you for the answers. Now I have obtained :

don't understand why I have the columns called unnamed.

My objective is to get a csv file that contains only the columns compiler and opt. How could I do this? Thank's again.

[EDIT 2] I have solved my problem but if I open the csv file manually, so just click on the file, I have the following:

s it contains only 11 lines, but it contains 3000 lines in jupyter. Is it there something wrong?

[EDIT 3] I tried to to the following:

import pandas as pd

test = pd.read_csv('1495927.csv')
test

and I have:

where 1495927.csv is the csv file I have created and that is in the image where I have 11 elements.


texts:
I would greatly appreciate if you could let me know how to use SMOTENC.  I wrote:

num_indices1 = list(X.iloc[:,np.r_[0:94,95,97,100:123]].columns.values)
cat_indices1 = list(X.iloc[:,np.r_[94,96,98,99,123:160]].columns.values)
print(len(num_indices1))
print(len(cat_indices1))

pipeline=Pipeline(steps= [
    # Categorical features
    ('feature_processing', FeatureUnion(transformer_list = [
            ('categorical', MultiColumn(cat_indices1)),

            #numeric
            ('numeric', Pipeline(steps = [
                ('select', MultiColumn(num_indices1)),
                ('scale', StandardScaler())
                        ]))
        ])),
    ('clf', rg)
    ]
)

Therefore, as it is indicated I have 5 categorical features. Really, indices 123 to 160 are related to one categorical feature with 37 possible values which is converted into 37 columns using get_dummies.

I think SMOTENC should be inserted before the classifier ('clf', reg) but I don't know how to define "categorical_features" in SMOTENC. Besides, could you please let me know where to use imblearn.pipeline?

Thanks in advance.


texts: How can I fill NaN values in a Pandas DataFrame in Python?

texts:
I am trying to learn data analysis and machine learning by trying out some problems.

I found a competition "House prices" which is actually a playground competition. Since I am very new to this field, I got confused after exploring the data. The data has 81 columns out of which 1 is the target column which is the house value. This data contains multiple columns where majority of values are "NaN". When I ran:

nulls = data.isnull().sum()
nulls[nulls > 0]

This shows the columns with missing values:

LotFrontage     259
Alley           1369
MasVnrType      8
MasVnrArea      8
BsmtQual        37
BsmtCond        37
BsmtExposure    38
BsmtFinType1    37
BsmtFinType2    38
Electrical      1
FireplaceQu     690
GarageType      81
GarageYrBlt     81
GarageFinish    81
GarageQual      81
GarageCond      81
PoolQC          1453
Fence           1179
MiscFeature     1406

At this point I am totally lost and I don't know how to get rid of these "NaN" values.
Any help would be appreciated.


texts: Transfer learning from great labelled time series data to one with low quality labelling

texts:
I have a source dataset containing outputs from a sensor per minute and have made extra effort to label them correctly for approx. 3 weeks.
I trained CNN-BLSTM network on that dataset which classifies with good accuracy.
Now I would like to use that knowledge to different datasets (very similar sensors in very similar environments), which have a good quality of sensor data, but the problem are the labels. There are 3 types of labelling problems:

target doesn't have any labels
target with lower quality labelling, e.g. instead of 5 categories they are categorized to 2 or 3
short periods are highly labelled

Can I use the transfer learning to all of these problems?
I guess highest Dense layer can be retrained with new data with different labelling categories?
Also, can I use the model with minute level data, but labelling in e.g. 15-minute bins?


texts: Should I eliminate all ID columns and similar columns from training data?

texts:
This is a basic question so bear my ignorance. I feel like they contribute collectively in no way to the target. This is for performance and accuracy.
The target is polar (0,1).


texts:
I finally developed a Game Bot that learns how to play the videogame Snake with Deep Q-Learning. I tried with different neural networks and hyper-parameters, and I found a working set-up, for a specific set of rewards.  The problem is is: when I reward the agent for going in the right direction - positive rewards in case the coordinates of the agent increase or decrease accordingly to the coordinates of the food - the agent learns pretty fast, obtaining really high scores. When I don't reward the agent for that, but only negative rewards for dying and positive for eating the food, the agent does not learn. The state takes into account if there's any danger in proximity, if the food is up, down, right or left and if the agent is moving up, down, right or left.
Here's the question: is rewarding the agent for going into the right direction a "correct approach" in Reinforcement Learning? Or it's seen as cheating, cause the system needs to learn that by itself? Is passing the coordinates of the food as state an other way of "cheating"?


texts: How do I cluster data that is a mix of text & categorical data?

texts: I have a set of strings, each also has soem categorical information associated with it. The categorical information isn't always great though, so I need to cluster the messages based on the text content & the categories. What is the best way to do this?


texts: Create a pipeline for recursive eliminartion using artifical neural nets

texts:
I am trying to use RFE with Artificial neural nets but I am getting the error that "'Sequential' object has no attribute '_get_tags'" . Here is my code snippet. Any help would be appreciated.
model_2 = Sequential([
    Dense(9, activation='linear'),
    Dense(200, activation=tf.keras.layers.LeakyReLU(alpha=0.3)),
    Dense(1, activation='linear'),
])

adam = keras.optimizers.Adam(lr=0.0001, beta_1=0.9, beta_2=0.999,epsilon=None, decay=0.0, amsgrad=False)
model_2.compile(optimizer=adam,loss='mse',metrics=['accuracy'])

rfe = RFE(model_2, n_features_to_select=5)
pipeline = Pipeline(steps=[('s',rfe),('m',model_2)])

hist_2 = pipeline.fit(X2_train.iloc[:,10:20].values,     y2_train.iloc[:,1].to_numpy(),
    m__batch_size=10, m__epochs=4000,
    m__validation_data=(X2_test.iloc[:,10:20].values, y2_test.iloc[:,1]))


texts: Balancing XGboost still skews towards the majority class

texts:
I have unbalanced dataset for multiclass classification and I tried to use the class weights option in XGboost and the classifier still tends to favor the majority class. I am not sure if I need to tune something else or how I should approach this.
If the algo was predicting all over the place I could still understand but not sure why it still skews towards the majority class. Any pointers?
Update: What I mean is that majority of the predictions are still for the major class which is roughly 30% in both test and train


texts: Cannot solve FrozenLake with QLearning and slippery mode enabled

texts:
I'm trying to solve the open-ai gym frozen lake problem with a basic Q learning table and it works quite well with is_slippery=False.
But it cannot find a solution with is_slippery=True because of the randomness. I decreased the epsilon decay, increased the number of iterations, decreased the max number of steps without any success.
The weird thing is that the best move on the first position is to go left which is obviously wrong because it goes outside the lake.
What am I missing?
Here's the final code
import numpy as np
import gym
from enum import Enum

epsilon = 0.99
min_epsilon = 0.01
epsilon_decay_rate = 0.9990
learning_rate = 0.81
discount = 0.96

total_episodes = 1000000
max_steps = 100

env = gym.make('FrozenLake-v0', is_slippery=True)

q_table = np.zeros([env.observation_space.n, env.action_space.n])

class ActionType(Enum):
  Exploration = 1
  Exploitation = 2

def get_next_action_type(epsilon):
  if np.random.uniform(0, 1) < epsilon:
    return ActionType.Exploration
  else:
    return ActionType.Exploitation

def reduce_epsilon(epsilon):
  epsilon *= epsilon_decay_rate
  return max(epsilon, min_epsilon)

def choose_action(state, epsilon, is_training=True):
  if is_training:
    action_type = get_next_action_type(epsilon)
    if action_type == ActionType.Exploration :
      return env.action_space.sample()
  return np.argmax(q_table[state, :])

def learn(state, new_state, reward, action):
  predict = q_table[state, action]
  target = reward + discount * np.max(q_table[new_state, :])
  q_table[state, action] = q_table[state, action] + learning_rate * (target - predict)

env.render()
env.reset()
for current_episode in range(total_episodes):
  state = env.reset()
  if current_episode % 500 == 0:
    epsilon = reduce_epsilon(epsilon)

  for step in range(max_steps):
    action = choose_action(state, epsilon)
    new_state, reward, done, info = env.step(action)

    learn(state, new_state, reward, action)

    state = new_state
    if done:
      #if reward == 1:
        #print("Episode {} finished after {} timesteps".format(current_episode, step + 1))
      break

np.set_printoptions(suppress=True)
print(np.argmax(q_table,axis=1))
print(q_table)

#LEFT = 0
#DOWN = 1
#RIGHT = 2
#UP = 3

and here's the result
Argmax of Q Table:
[0 3 3 3 0 0 2 0 3 1 0 0 0 2 1 0]
QTable:
[[0.63035982 0.25384749 0.25463065 0.40253981]
 [0.01489107 0.0492833  0.24330277 0.25612783]
 [0.14042617 0.16831635 0.1728443  0.25067629]
 [0.02342756 0.16342706 0.14269766 0.16964899]
 [0.5452489  0.25566138 0.51625068 0.36098437]
 [0.         0.         0.         0.        ]
 [0.04127257 0.         0.36224538 0.00009004]
 [0.         0.         0.         0.        ]
 [0.28458546 0.07837945 0.0009689  0.57059009]
 [0.11749986 0.67078974 0.02244687 0.32798165]
 [0.40534747 0.1290854  0.12926799 0.33130133]
 [0.         0.         0.         0.        ]
 [0.         0.         0.         0.        ]
 [0.49669734 0.17589801 0.89115956 0.65811115]
 [0.62889188 0.97518509 0.48873376 0.53546004]
 [0.         0.         0.         0.        ]]


texts: How do I allow user to run a R script without making a website or web application?

texts:
I have data & a R script that creates a report from the data. I can't expose the data to internet. Also, I cant expose my script to internet / user. But I would like to eliminate myself from the work, and allow couple of users (yeah, only three users that use this script to generate reports, but they do it weekly) to run the script and generate report for them.
I would like the community to suggest me how I can achieve this. Any information helps.


texts:
Why is covariance considered as inner product if there is no projection of one vector onto another?

Right now I perceive this as just a multiplication of $x$ segment of vector($x_i - \bar{x}$) and $y$ segment($y_i - \bar{y}$) of the same vector in order to understand direction of relationship.


texts: Normalisation results in R^2 score of 0 - Lasso regression

texts:
I am running a regression analysis on a 7000 row dataset with a train/test split of 70%/30%. I am using one variable X to predict a variable Y.

X ranges between 300 and 810 (mean 712).
Y is an integer (number of occurrences) ranging between 0 and 20 (mean 0.2).

Without standardisation or normalising X, I receive:

Train score:  0.082
Test score:  0.077

However upon normalising (X = (X-X.min())/(X.max()-X.min())), I receive:

Train score:  0.0000
Test score:  -0.0001

Is there something incorrect about normalising for a Lasso regression? The same applies to standardising the data. Would anyone be able to advise me on the best course of action?


texts: How to deal with with rows with zero in every feature while clustering?

texts:
I am working on a clustering problem which has 13000 observations and 15 features. Around 3000 observations in the dataset has  zero in every features ( i.e all values zero in 3000 rows). I am trying to do clustering on top of it. What is a better way to deal with it ? I have few things in my mind but would like to get clarity on :

Check for number of rows with all zero and remove them ?
Include the rows with zero value in every feature and let the clustering
algorithm handle the same?

Also,please suggest if there are any better way to handle the same.

Note : I am using k-means clustering.


texts:
I have used the same methods/parameters to create two decision trees. The trees classify the presence or absence of a medical condition using the presence or absence of various symptoms. There is a tree for Medical Condition #1 and another tree for Medical Condition #2. Both trees are based on the same set of symptoms, rated by patients. If Medical Condition #1 resulted in a much simpler tree than Medical Condition #2, can that suggest Medical Condition #2 is a more complex disease? If so, can anyone point me to a reference that states the complexity/depth of a tree can be representative of a complex condition?

The goal of my analysis is separate from this question. I am trying to figure out all the conclusions I can draw or suggest from my analysis. Yes, I am interested in saying something about the complexity of condition A to B.
When is a condition complex ? When there are many symptoms as the disease is diagnosed by the symptoms.
If it's hard to diagnose ? YES
if the symptoms are severe ? NO
can s.o. have both conditions ? NO


texts: Embedding dimension size for a custom Word2Vec?

texts:
Are there any guidelines for choosing the embedding dimension size value in a custom Word2Vec embedding? I know that the default is 100 and that seems just as good as any. But I'm wondering if there is any data out there that creates heuristics from when you should deviate from this value.

I can't imagine that there is much benefit in a smaller size, but there must be some value in a larger size? Or maybe it's related to the size of my vocabulary? I have a relatively small vocabulary for my latest project (7,000 words) so maybe there is some ratio or proportion that I can apply?


texts:
Model blending -- by which I mean creating multiple sets of predictions from models that have the same dependent variable and the same or similar independent variable candidates, as opposed to model stacking -- is a popular way of creating ensembles of Machine Learning models.

For example:

Y = regression_predictions * .5 + tree_predictions * .5

While this approach is useful in a wide variety of use cases, a good example is Kaggle competitions. In these competitions you have:

Labeled training data
Unlabeled prediction (test) data
A certain number of allowed submissions per day, often 5, wherein you submit your predictions and receive instant scoring based on a specified evaluation metric like RMSE

My question is if there is an statistical or logical methodological framework that can guide the testing of various blends of your models?

That is, can some algorithm or methodology be applied to the process of deciding how to alter the blend of predictions?

Ideally, such a framework would also guide the decision of when to add or remove a set predictions, though even a framework which assumes a given number of models would be highly useful.


texts: Database options for JSON storage, queried with Apache Drill

texts:
I am planning to set up a JSON storage system. It will store tens of millions of JSON records, all in the same format. I'd like to be able to query the data using Apache Drill.

It looks like there is Drill support for MongoDB and Postgres. However, I'm unsure of the pros and cons of each, and how I'd structure the schema if I'd choose Postgres.


texts: Cost function for Ordinal Regression using neural networks

texts: What is the best cost function to train a neural network to perform ordinal regression,  i.e. to predict a result whose value exists on an arbitrary scale where only the relative ordering between different values is significant (e.g: to predict which product size a customer will order: 'small' (coded as 0), 'medium'(coded as 1), 'large' (coded as 2) or 'extra-large'(coded as 3))? I'm trying to figure out if there are better alternatives than quadratic loss (modeling the problem as an 'vanilla' regression) or cross-entropy loss (modeling the problem as classification).


texts: Normalizing a list with both positve and negative numbers to the range [-1 1]

texts:
For example, I have a list [-518.8134, 480.1884,160.4761]
How to normalize this to [-1 1] range?


texts: How to incorporate the "current time" into an parcel ETA prediction model

texts:
I need to create a model that is able to predict the ETA of parcels, using only discrete events about when and where the good was last processed.
For each parcel I have a sequence, of varying length, denoting the events of that parcel. Events such as "arrival to a distribution" center or "ready for pickup". All events have a timestamp and location.
My current approach has been to split up each sequence, and treat each event independently (assuming that the time it takes to go from B -> C is not dependent on how we arrived at B). I have trained a random forest regressor, that given an event and the location the parcel needs to go, is able to somewhat accuractly predict the ETA.
The problem is that the model takes events as input, and predicts the relative time in hours from that event to the ETA. This means that, if the model is wrong, and the parcel hasn't been delivered by that ETA, I would like to provide a new ETA, but as no new events has happend to that parcel, the model would just predict the same wrong ETA.

How do I incorpoorate the current time into the model?
What is this type of problems called?
What other type of models would be more suitable for these type of problems?

Thanks in advance.


texts: Spark fitting into Data Science Paradigm for timeseries data

texts:
I have seen Dataframe as new API on Spark2 instead of RDD.So I have following few question about the utility of Spark in terms of time series data.

Is forecasting still limited to memory available in one computer
instead of utilizing distributed fashion of Spark?
Has storing time series data in form of Dataframe rather than RDD
eliminated the necessity of creating separate TimeseriesRDD
which some projects did such as

Sparkts library

Flint library
How is processing of TBs of data utilized in forecasting and
deciding models such as ARIMA, ARMA if the first question is true?


texts: Why could an overfitted CNN model have a higher validation accuracy?

texts:
I am currently training a CNN model by using cifar10 images (50000 for training, another 10000 for validation).

I plot training loss, validation loss and accuracy against training iteration:

I am not sure when should I stop the training, should I stop it at around iteration 1250, since overfitting happens beyond that point? Or should I stop it at around iteration 5000, since I can get the maximum validation accuracy?

Why can the overfitted model have a higher validation accuracy?


texts: What can be the best dropout value and the FC layer for good accurate predictions?

texts:
I am retraining the pre-trained model  VGG16 in the last FC layers.
I used the below function .

what can be the best combination of FC layers and dropout values for the best predictions. ?


texts: What is the difference between Perceptron and ADALINE?


texts: Multidimensional K-Means wiith sklearn, centroids problem when plotting

texts:
I am working with a dataset (X) to predict 12 clusters with K-Means using python SKLEARN library:
numClusters= 12
kmeans = KMeans(n_clusters=numClusters).fit(X)
centroids = kmeans.cluster_centers_

# Predicting the clusters
labels = kmeans.predict(X)
# Getting the cluster centers
C = kmeans.cluster_centers_

#transform n variiables to 2 principal components to plot
pca = PCA(n_components=2)
principalComponents = pca.fit_transform(X)
principalDf = pd.DataFrame(data = principalComponents
         , columns = ['principal component 1', 'principal component 2'])

colors =['red','green','blue','cyan','yellow', 'lime','orange','coral','brown','peru','khaki','tan']
centroidColor= []
for item in range(numClusters):
  centroidColor.append(colors[item])

dataPointColor=[]
for row in labels:
  dataPointColor.append(colors[row])

fig = plt.figure(figsize = (10,10))
ax = fig.add_subplot(1,1,1)
ax.set_xlabel('Principal Component 1', fontsize = 15)
ax.set_ylabel('Principal Component 2', fontsize = 15)
ax.set_title('2 component PCA', fontsize = 20)
plt.scatter(principalDf['principal component 1'], principalDf['principal component 2'],
c=dataPointColor, s=50, alpha=0.5)

plt.scatter(C[:, 0], C[:, 1], c=centroidColor, s=200, marker=('x'))
plt.show()

My problem comes when trying to plot the n dimensions of my dataset into 2 PCA. The clustering seems fine, but the centroids (marked with a star icon) are not correct:

The centroids should not always be in the same near spot.
I know that I did not standardize the dataset (X) because all values are in a range [1:5]:
[[4 5 4 ... 4 4 4]
[4 4 1 ... 4 3 5]
[3 4 3 ... 1 1 4]
...
[2 5 1 ... 2 1 4]
[5 5 1 ... 5 5 1]
[2 5 1 ... 1 1 5]]

Could the problem come from that?
Thanks for your help.


texts: How to Make Meaningful Conclusions here?

texts:
I recently appeared for an Interview for my college and I was asked the following question. The Interviewer said that this question was a Data Science question.
The question-
Suppose 7.5% of the population has a certain Bone Disease. During COVID pandemic you go to a hospital and see the records. 25% of the COVID Infected patients also had the Bone Disease. Can we say for sure if the Bone Disease is a symptom of COVID-19?
My Reponse-
I said No, and explained it as it's not necessary that COVID-19 is causing these symptoms, it could very well be possible that the 7.5% of the country's population which already had the disease is more susceptible to the virus due to lowered immunity. Hence making conclusions is not possible.
Then the interviewer asked me How can we be sure if it is a symptom or not?
I replied saying we can go to more Hospitals, collect more data and see if it correlates everywhere.
The Interviewer then said If we have the same results everywhere will you conclude it's a symptom?
I had no good answer but I replied that Just correlation of data is not sufficient, we also need to check if the people who have COVID-19 had the bone disease prior to getting infected or not. See if that percentage also correlates and stuff.
Here he stopped questioning however I couldn't judge If I was right or wrong.
I am in Grade-12 so I have no experience in Data Science as such. I do know a fair bit of statistics however I have never solved such questions. Can someone provide me insights on how to solve such questions and make meaningful conclusions?


texts: Feature Selection Statistical Test for Nominal Response Vs Continuous Predictors? (in R)

texts:
I cannot find much information on this, none so far useful.
I have a sparse data set with 17K+ columns of continuous gene expressions, an example of a typical column: (3.15, 0, 7.1294, 0, 0, 0, 2300.2, 213.444,...)
and each row has an associated nominal response of celltype = 'A', 'B', 'C', 'D', 'E'. These are names, hence nominal, and not ordinal.
Thus, there are 17308 gene expression columns, and a single response column in my data frame, with 4954 rows/samples.
I am trying to find the right statistical test to perform a filter method for feature selection (taking the best 1000 low-correlation p-values obtained through some statistical test). My teacher mentioned Wilcoxon rank-sum test, but I don't see how this applies, nor ANOVA. I found the Kruskal-Wallis as the only relatively good test. I do not come from a statistics background, having only taking Mathematical Statistics I in university.
I use a File Backed Matrix (FBM from bigstatsr package) to assign the p-values in a foreach loop, as follows:
foreach(i = 1:num_feats, .combine = 'c') %dopar% {
  dat_total = cbind(df[, i], df$ctype)
  p_vals[i - 1, 1] <- kruskal.test(df[, i] ~ df$ctype, data = dat_total)$p.value
}

but this results in about 12000 p_values stacked up at zero... (?)
I also tried switching  df[, i] ~ df$ctype for df$ctype ~ df[, i] but the kruskal.test in R asks for response ~ group, regardless, when I do this I get a bimodal distribution of p-values: a few thousand stacked up near 0 (but non are zero) and a few thousand stacked up near 1 (some equal to 1). I read more on this as in 'Section C' found here.

I'm supposed to just test each column against the response, check for significance, peel off the best 1000, and then generate a feature reduced matrix df_sml that I then feed into the elastic net, lasso, linear discriminant analysis, and a classification tree. What I did, however, was pick the best 1645 p-values, and pick the top 1000 that had a correlation between -0.25 and 0.25 (LDA was failing otherwise).
ANY tips are appreciated, I don't see the point of deriving a test data set for reproduction. This(and several other) articles mentioned Kruskal-Wallis for categorical dependent versus continuous independent variables (but my concern is I am feeding them in 'backwards' to the test in R).
I don't understand why I would possibly want to factor cell type into a dummy variable, as this is nominal data and not ordinal data, so any explanation regarding that appreciated. If it is okay, why? Doesn't ordinal data imply rank? And if I am treating the dummy variable as a factor, how is that different than treating 'A', ..., 'E' as factors?
The simplest method you can suggest would be most appreciated, hence my turn to Kruskal-Wallis and not some of the more complicated and involved methods I have seen suggested (I likely wouldn't have the time to implement them).
edit: I wanted to add a few things: I ran elastic net for alpha = 0.1:0.9 by 0.2 and LASSO, and I am getting a classification accuracy rate of around 0.98 for all of these. So unless something is amiss, my models are getting pretty good results, LDA performed just slightly worse (0.96) than Elastic Net (glmnet), the Tree model with pruning (0.76) was garbage lol. And this was using df$ctype ~ df[, i] which is group ~ response, not the desired format by kruskal.test as far as I know. {see ?kruskal.test(formula,...)}
edit2: I wanted to add, there are like 8K+ columns with 95% entries being zero, but the teacher said to leave them in and do this test instead (this data is for a machine learning course midterm project), which I've read can be what is contributing to the wild number of p-values stacking up at or near 1. I originally stripped these columns from the data frame.


texts: What is the difference between trax vs tensorflow?

texts:
What is the main difference between trax vs tensorflow?
both of them deep learning library and implemented by google team.
https://github.com/google/trax
https://github.com/tensorflow/tensorflow


texts: Scikit MLPClassifier vs. Tensorflow DNNClassifier

texts:
I am trying to understand the differences between Scikit MLPClassifier and Tensorflow DNNClassifier for classification task and hoping that some experts can share a light. as far as I understand, both supports backpropagation, activation functions (inc. relu), optmizer (sgd/adam). MLPClassifier can also have deep neural networks by specifying the # of hidden layers and nodes.  The only difference between two that I can see is DNNClassifier supports GPU training while MLPClassifer does not. Outside of GPU support, are there other differences between them? Why do one want to use Tensorflow's DNNClassifier for basic deep neural network training (I am talking basic feed-forard NN here not the CNN, RNN, LSTM, etc.) than Scikit's MLPClassifier?

Thanks!


texts: Machine learning with sklearn vs. scipy stats

texts:
I've created 50 random x and y points (with slope of y = 2x-1).

First, I used Linear Regression from sklearn to fit the model onto my dataset where I got a slope of 2.0066... and intercept of -0.535...

My Question: is fitting the model to our dataset considered training? For each given x value, since it has a y-value (supervised), does our machine go through each x,y match and create line of best fit based upon that? Thus, is our model trained?

Second, I used stats.linregress(x,y) from scipy to get slope and intercept (which seem really close if not the same to the slope and intercept I've got from using sklearn Linear Regression).

My Question: If both methods give the same result, why not just use scipy to get formula for the best fit line to make predictions? What is the benefit of using machine learning?


texts:
I would like to ask you how to use classifier and determine accuracy of models.
I have my dataset and I already cleaned the text (remove stopwords, punctuation, removed empty rows,...).
Then I split it into train and test.
Since I want to determine if an email is spam or not, I have used the common classifiers, I.e. Naive Bayes, SVM and logistic regression.
Here I just included my train and test datasets: nothing else!
I am using Python to run this analysis.
My question is: should it be enough or should I implement new algorithms?
If you could provide me with an example of how an already existing algorithm was improved it would be also good.
I read a lot of literature regarding accuracy of text classification and in all the papers the authors use SVM, Naïve Bayes, logistic regression to classify spam.
But I do not know if they built their own classifier or just used the existing one in Python.
Any experience on this?


texts:
Let's say I have a huge database with 100K records and 60 columns. Let's say one of the column is "min_p". What I do is apply some logic/rule to determine the output label for this record. Basically I look at previous two records and next two records of this min_p. If the condition is satisfied, I will mark the label as 1 else I will mark it as 0.

Now my question, since I have directly derived the label from this called "min_p", should I retain it as one of my predictors in my final dataset? Since I have used that derive the label,  I didn't include them in my dataset as a input variable thinking that it is incorrect

Can you help me with this?


texts:
Just getting my toes wet with running some models on the Iris dataset and was wondering if using One-vs-Rest is required or not? Because I can fit a linear model without it, but using OVR yields different results.

Any explainations would be great, thank you!


texts: Can CBOW model only accept fixed number of words?

texts: I have a question about CBOW prediction. Suppose my job is to use 3 surrounding words w(t-3), w(t-2), w(t-1)as input to predict one target word w(t). Once the model is trained and I want to predict a missing word after a sentence. Does this model only work for a sentence with four words which the first three are known and the last is unknown? If I have a sentence in 10 words. The first nine words are known, can I use 9 words as input to predict the last missing word in that sentence?


texts: How do I provide input and output for such a network structure in keras

texts:
I am trying to create a CNN network for classification purposes, the network with both input and output is illustrated as such:

The input the image is separated into sections, each section is given to a certain model in which a convolution and pooling is applied. This is done as such to ensure full weight share does not occur, as I am not interested in the complete image shares the same weight.  After the convolution an pooling, is all the output of models concatenated into one, and fed into a fully connected layer, which then performs classification and outputs a class.

The input I have is the image (either the complete or separated into sections) and the output being the class, but I am not sure how I should train the model, given these input and output.

The way I've constructed the model is as such:

#Convolution for different section conv2d_[convolution#]_[model#]
conv2d_1_1 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_1)
conv2d_3_1 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_1)
conv2d_4_1 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_1)
conv2d_4_1_flatten = Flatten()(conv2d_4_1)

conv2d_1_2 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_2)
conv2d_3_2 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_2)
conv2d_4_2 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_2)
conv2d_4_2_flatten = Flatten()(conv2d_4_2)

conv2d_1_3 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_3)
conv2d_3_3 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_3)
conv2d_4_3 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_3)
conv2d_4_3_flatten = Flatten()(conv2d_4_3)

conv2d_1_4 = Conv2D(filters = 32, kernel_size = (3,3) , padding = "same" , activation = 'relu' , input_shape = (3,6,3))
conv2d_2_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_1_4)
conv2d_3_4 = Conv2D(filters = 64, kernel_size = (3,3) , padding = "same" , activation = 'relu' )(conv2d_2_4)
conv2d_4_4 = Conv2D(filters = 32, kernel_size = (1,1) , padding = "same" , activation = 'relu' )(conv2d_3_4)
conv2d_4_4_flatten = Flatten()(conv2d_4_4)

#Merging the output of the convolution
merge = Merged([conv2d_4_1_flatten, conv2d_4_2_flatten,
                conv2d_4_3_flatten, conv2d_4_4_flatten] mode = 'concat')

#Connecting the merged layer to the fully connected layer
dense1 = Dense(activation = 'relu')(merge)
dense2 = Dense(activation = 'relu')(dense1)
dense3 = Dense(output = 1 ,activation = 'softmax')(dense3)

This (I guess) CNN model should now be connected, but how do I train it given the input and outputs I have? How do I parse it?

My input data is stored as list of numpy.ndarray. Each row of each ndarray, is the input for each input.  How do I parse each row to each input? And the output is a vector of classes. I have in total 145 classes, and output is vector length 3 with the true labels extracted from 4 input channels?


texts: SMOTE for multi-instance learning i.e num_rows(x_train) > num_rows(y_train)

texts:
I have an imbalanced dataset and I wish to predict classes(0 or 1).

Sample x_train:

id      date    c1   c2 . . . . . .  c20
101  13-02-2015  2    7 . . . . . .   14
101  14-02-2015 24    7 . . . . . .    8
  .
  .
  .
105  13-02-2015 12    5 . . . . . . .  4
  .
  .

Sample y_train

id   class
101    1
105    1
107    0
 .
 .
 .

Now I wish to over sample class 0 in the dataset but the problem is for each id I have just one row in y_train whereas I have 50 rows for the same id in x_train.


texts: How many Layers in the original Yoon Kim CNN implementation?

texts:
I saw some implementations of Yoon Kim's Convolutional Neural Network (Paper: http://www.aclweb.org/anthology/D14-1181)....

...in some implementations they put one more Dense(..) Layer before the output layer (with softmax activation). In my opinion there is no additional Dense(..) Layer in the original paper:

So what's the original implementation?


texts: How to use the same minmaxscaler used on the training data with new data?

texts:
Im using the keras LSTM model to make prediction, and the code above is to scale the data:
inputs are shaped like  (n, 11, 1) and the label is 1D
DailyDemand.py

#scaling data
scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
x = np.array(x).reshape ((len(x),11 ))
x = scaler_x.fit_transform(x)
scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
y = np.array(y).reshape ((len(y), 1))
y = scaler_y.fit_transform(y)

# Split train and test data
x_train=x[0: train_end ,]
x_test=x[train_end +1: ,]
y_train=y[0: train_end]
y_test=y[train_end +1:]
x_train=x_train.reshape(x_train.shape +(1,))
x_test=x_test.reshape(x_test.shape + (1,))
# Train and save the Model named fit1 in a json and h5 files
[....]
# serialize model to JSON
model_json = fit1.to_json()
with open("model.json", "w") as json_file:
    json_file.write(model_json)
# serialize weights to HDF5
fit1.save_weights("model.h5")
print(">>>> Model saved to model.h5 in the disk")

and Now im trying to predict a new values of a new data with this trained model. so i loaded the model from the files:

predict.py

from DailyDemand import scaler_y
from DailyDemand import scaler_x
[...]
# load json and create model
json_file = open('model.json', 'r')
loaded_model_json = json_file.read()
json_file.close()
loaded_model = model_from_json(loaded_model_json)
# load weights into new model
loaded_model.load_weights("model.h5")
print("Loaded model from disk")

########################################
# make prediction with the loaded model

FeaturesTest = [267,61200,695,677,70600,116700,130200,768,659,741,419300]
xaa = np.array(FeaturesTest).reshape ((1,11 )).astype(float)
print(xaa)
xaa = scaler_x.fit_transform(xaa)
xaa = xaa.reshape(xaa.shape +(1,))
print("print FeaturesTest scalled: ")
print(xaa) # incorrect scalled value, always returns -1 ones

xaa = [[[-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]
  [-1.]]]

tomorrowDemand = loaded_model.predict(xaa)
print("tomorrowDemand scalled: ", tomorrowDemand)
prediction = scaler_y.inverse_transform(np.array(tomorrowDemand).reshape ((len(tomorrowDemand), 1))).astype(int)
print ("the real demand is 95900 and the prediction is: ", prediction)

The problem is how i can use the same scaler used in the training on the new data?
i want to know if i made a mistake in this code to use the same scaller on the new data?


texts: How to replace NaN values for image data?


texts: number of parameters for convolution layers

texts: In this highly cited paper, authors give the following discussion on the number of weight parameters. I am not very clear why it has $49C^2$ parameters. I think it should be $49C$ since each of $C$ input channels shares the same filter, which has $49$ parameters.


texts: How should I collect data for an "identifier" rather than classifier?

texts:
I'm trying to build a CNN to identify whether or not a picture contains a lion. One of my classes would obviously be filled with pictures of lions, and pictures that have lions.

What about the data I put in my not_lion class? Should I fill it with images of random things (for example, take a few images from each class in ImageNet)? Should I put in pictures of things similar to lions, like tigers, so the network better learns what a lion actually is?

My idea is to make 50% of the not_lion class pictures of tigers, pumas, etc, and 50% random stuff - I think that this would make the network learn what a lion actually is, and also ensure the network doesn't classify a laptop as a lion.


texts: Significant drop from validation accuracy to test accuracy

texts:
I am more familiar with classification tasks, though I have been working on a regression problem. I was given a large training dataset (>70k samples) and an independently collected test set (~2k). I consistently achieve decent validation accuracy but significantly lower accuracy on the test set.

I've been performing validation like this:
1) Standardize the training data; store the mean and variance (in an sklearn Scaler object).
2) Hold out 10% training data for validation.
3) Perform 3-fold cross validation on the remaining 90% training data. Look at the errors.
4) Train the model on the 90% training data (with 10% of this subset used as validation during training).
5) Evaluate the performance on the 10% held-out training data. This is used as my final validation measure.

Then I test my model by:
1) Standardizing the test data using the mean and variance of training data.
2) Predict and evaluate

Depending on the model I test, I typically receive 10+% drop in accuracy from validation to test. Now, to test whether this is caused by the test set coming from a different distribution, I combined the test set (~2k) with an equal-sized random portion of the training set. Then I held out 10% of this as a new test set and performed training/validation as described above using the rest. So in this case the test set is no longer an independently collected dataset.

It appears that I'm in some way overfitting to the validation set, although as I describe above, the validation portion is held out until after training.

I think it is worth noting that my target variable and many predictors are exponential-like distributed (the target variable has a lower bound at 0 and increasingly rare large values). Heteroscedasticity is apparent because plotting predicted vs. observed output shows increasing error with larger values.

Here are results using an MLP regressor with the original training data and test data (Left: Validation, Right: Test):

Here are results using an MLP with equally combined train/test (~4k) as training, with 10% held out for testing (as described) (Left: Validation, Right: Test):

Here are results using a Random Forest with same train/test as for the figure directly above (Left: Validation, Right: Test):

I would greatly appreciate any ideas of the cause of this discrepancy. Thank you!


texts:
I am using some python code from kaggle that plots a bar graph. However, when I run it, it does not display the graph. When I run it in the IDE I get the following output:

<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>
<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>
<matplotlib.axes._subplots.AxesSubplot object at 0x1200b9ba8>

I have used the .show() method but nothing seems to work.

Here is the link to the code that I am using:

Code

I also have it directly in the question:

import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import matplotlib
from sklearn.linear_model import LinearRegression

# Input data files are available in the "../input/" directory.
# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory

#from subprocess import check_output
#print(check_output(["ls", "../input"]).decode("utf8"))

# Any results you write to the current directory are saved as output.

frame=pd.read_csv('Project_Airplane_Crashes_and_Fatalities_Since_1908.csv',sep=',')
matplotlib.rcParams['figure.figsize'] = (12.0, 8.0)

operator = frame[['Operator','Fatalities']].groupby('Operator').agg(['sum','count'])

fig_ops,((ax1, ax2), (ax3, ax4))=plt.subplots(2,2,sharex=True)
accidents = operator['Fatalities','count'].sort_values(ascending=False)
totalacc = accidents.sum()
accprop = accidents/totalacc
accidents.head(10).plot(kind='bar',title='Accidents by Operator',ax=ax1,grid=True,rot=90)
accprop.head(10).plot(kind='bar',title='Proportion of Total Accidents',ax=ax2,grid=True,rot=90)

fatalities = operator['Fatalities','sum'].sort_values(ascending=False)
fatalities.head(10).plot(kind='bar',title='Fatalities by Operator',ax=ax3,grid=True,rot=90)
totalfatal = fatalities.sum()
fatalprop = fatalities/totalfatal
#print(fatalprop)
fatalprop.head(10).plot(kind='bar',title='Proportion of total Fatalities',ax=ax4,grid=True,rot=90)

Any help would be greatly appreciated.


texts: Keras - How to using Cnn trained model on video image.?

texts:
I have trained Cat and dog image using CNN with Python ( total 10K, 8K for training and 2K for testing ).
I want to make prediction in a video. Video Contain both cat and dog at a single moment. How can I do that. ?
How to make bounding rect with text on cat and dog image in video.
Thanks for helping.


texts: How to use precomputed distance matrix and min_sample for DBSCAN clustering method?

texts:
I want to perform DBSCAN on my datapoints, but I don't have access to the data, I just have the pairwise distance of datapoints. Additionally, I have no idea about the number of clusters but I do want that each cluster contains at least 40 data points. Does DBSCAN work with these conditions? For instance, can I have something like this? Or is more information needed? I want to emphasize that I have computed the pairwise distance and this is not the result of Euclidean or some other method.

from sklearn.cluster import DBSCAN

db = DBSCAN(min_samples=40, metric="precomputed")

y_db = db.fit_predict(my_pairwise_distance_matrix)

I am not sure what is eps parameter of DBSCAN(). How should I set that?


texts: Where can I get a comprehensive criminal dataset?

texts:
I want to create a machine learning model to predict the probability of Person committing crime in future. Given his/her past crime history, age, gender, race, employment history, family information (Married, Having young kids, etc.), financial status etc.

Please suggest some data sources having such information to train the model.


texts: Is reinforcement learning a subset of unsupervised learning?

texts:
According to this article:

Reinforcement learning on the other hand, which is a subset of Unsupervised learning ...

How true is this statement? Is there any scholarly discussion/writing on the classifications of Reinforcement learning?


texts: How to preprocess data for multiple rows instance?

texts:
I have a machine learning problem where I must detect anomalies in (let say) taxes declaration. So I have multiple rows for each enterprise, each row describing the tax declaration a the time (date,value,...).
NB: the number of tax declaration is not the same for everyone (10 to 100).
How can I preprocess this data to make machine learning algorithm understand it?


texts: Can Nonnegative Matrix Factorization be used for extracting the most important features from a dataset with high dimensionality (1000+ variables)?

texts: I would like to extract and identify the most important variables from a biological dataset containing mutational and gene expression data in R. Previous supervised techniques showed that these features have weak predictability. Can I use NNMF for this aim?


texts:
I ran this code and this doesn't work, I'm using python 3 btw, I have checked the syntax a million times. I have installed all the necessary packages and all of them are up to date, here is the code I ran:

from sklearn import tree
features = [[140, 1], [130, 1], [150, 0], [170, 0]]
labels = [0, 0, 1, 1]
clf = tree.DecisionTreeClassifier()
clf = clf.fit(features, labels)
print(clf.predict([[150, 0]]))

Here is the console error message (I don't know what it's exactly called, please tell me if you know):

pydev debugger: starting

Traceback (most recent call last):
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\ptvsd_launcher.py", line 111, in <module>
    vspd.debug(filename, port_num, debug_id, debug_options, run_as)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\debugger.py", line 36, in debug
    run(address, filename, *args, **kwargs)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\_main.py", line 47, in run_file
    run(argv, addr, **kwargs)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\_main.py", line 98, in _run
    _pydevd.main()
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\pydevd.py", line 1628, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\pydevd.py", line 1035, in run
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\Extensions\Microsoft\Python\Core\Packages\ptvsd\pydevd\_pydev_imps\_pydev_execfile.py", line 25, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:\Users\Sanjay\Documents\python files\SLNforVt\VisualTest\VisualTest.py", line 6
    print(clf.predict([[150, 0]]))
                                  ^
SyntaxError: invalid character in identifier

I am using Visual Studios here, I do not know if that affects this program in anyway but I also tried it using the python idle. Other Python programs I write work fine on Visual Studios without any errors.


texts: what is criterion in flcuster of scipy package?

texts: Could some one explain what does criterion of fcluster indicate? I tried to read the documentation but I am unable to understand. What does maxclust criterion indicate?


texts: Model selection Tensorflow for custom dataset comparing panoramic images vs regular images (Image segmentation)

texts:
I have a question regarding the use of a specific model such as Deeplab and how to create a custom dataset for it.
Background
To give a bit of background info, I want to compare panoramically stitched images of 360 degree (consisting of 6 stitched images) vs the same 6 images without stitching. Obviously you lose information by stitching these images into a single panoramic image since it contains much less pixels but it gives you a much better user experience. My goal is to find how much accuracy I lose in comparison to the speed and user experience I gain. My problem however is the lack of data, therefore I want to use a pre-trained model.
Dataset
I have created a custom dataset with labelled panoramic images and labelled the 6 images of the same situation to be able to compare them equally. This dataset contains 30 panoramically labelled images and 180 regular labelled images with the same classes as Cityscapes (https://www.cityscapes-dataset.com/). Below shows an example of one of my labelled panoramic image.

Problem
The problem with using a pre-trained model as Cityscapes is that it is trained on straight lines, while panoramic images are spherical images placed on a 2d plane causing lines to be round and not straight. The 6 seperate images on their own are straight and will always achieve better accuracy. Basically a pre-trained model such as deeplab or any other model is biased towards my 6 images making it unfair to compare them. Therefore I want to use my custom dataset and build it on top of an existing model such as deeplab or create my own simple model.
I don't have a lot of experience with machine learning, but it feels like Deeplab is one of the only pre-trained models that I can use for my issue. I have also thought about making my own model, but I am afraid that my lack of data makes it impossible to compare them (eventhough the accuracy in my case doesn't matter, I am looking for the difference in accuracy).
My first question is: Is Deeplab the correct model to use in this situation? Or should I go with a custom model? Or are there any other models that I can use for this situation?
Custom dataset
Besides the model selection, how can I make my dataset ready for a model such as deeplab in tensorflow? All the examples that I found on deeplab make use of the same 3 datasets: ADE20K, Cityscapes and Pascal VOC 2012. None of them show examples of custom datasets.


texts: My parents are elderly and a fall is a BIG DEAL.  I'm pretty good at coding and such.  So I thought I'd use some fall datasets and plug them into a machine learning system.  As you can see from this statement, I am new to machine learning.  How should I start getting my first model?  Literature has pointed me to trying a decision tree model as better than KNN...do I pick up TensorFlow and start plying a model with datasets, or how would you start?  Thank you and I apologize for my noob question.


texts:
I am trying to implement a Reinforcement Learning Algorithm in a trading scenario.
It seems natural to me to use final profit as the reward. However, in this scenario, for each trading episode, there is only one reward, obtained at the end. Is there any RL algorithm that could learn only based on final reward ?
If it was a mario game, how could I train a network only based on a reward earned at the end of the level, but that depends of all previous actions ?

What I'm trying to do is very similar to https://dzone.com/articles/trading-strategies-using-deep-reinforcement-learni


texts: ANOVA RBF kernel returns very poor results

texts:
I was curious about the ANOVA RBF kernel provided by kernlab package available in R.

I tested it with a numeric dataSet of 34 input variables and one output variable. For each variable I have 700 different values. Comparing with other kernels, I got very bad results with this kernel.
For example using the simple RBF kernel I could predict with 0,88 R2 however with the anova RBF I could only get 0,33 R2.
I thought that ANOVA RBF would be a very good kernel. Any thoughts? Thanks

The code is as follows:

set.seed(100) #use the same seed to train different models
svrFitanovaacv <- train(R ~ .,
                       data = trainSet,
                       method = SVManova,
                       preProc = c("center", "scale"),
                       trControl = ctrl, tuneLength = 10) #By default, RMSE and R2 are computed for regression (in all cases, selects the tunning and cross-val model with best value) , metric = "ROC"

define custom model in caret package:

library(caret)
#RBF ANOVA KERNEL
SVManova <- list(type = "Regression", library = "kernlab", loop = NULL)
prmanova <- data.frame(parameter = c("C", "sigma", "degree", "epsilon"),
                     class = rep("numeric", 4),
                     label = c("Cost", "Sigma", "Degree", "Epsilon"))
SVManova$parameters <- prmanova
svmGridanova <- function(x, y, len = NULL) {
library(kernlab)
sigmas <- sigest(as.matrix(x), na.action = na.omit, scaled = TRUE, frac = 1)
expand.grid(sigma = mean(sigmas[-2]), epsilon = 0.000001,
            C = 2^(-40:len), degree = 1:2) # len = tuneLength in train
}
SVManova$grid <- svmGridanova
svmFitanova <- function(x, y, wts, param, lev, last, weights, classProbs, ...) {
  ksvm(x = as.matrix(x), y = y,
       kernel = "anovadot",
       kpar = list(sigma = param$sigma, degree = param$degree),
       C = param$C, epsilon = param$epsilon,
       prob.model = classProbs,
       ...) #default type = "eps-svr"
}
SVManova$fit <- svmFitanova
svmPredanova <- function(modelFit, newdata, preProc = NULL, submodels = NULL)
  predict(modelFit, newdata)
SVManova$predict <- svmPredanova
svmProb <- function(modelFit, newdata, preProc = NULL, submodels = NULL)
  predict(modelFit, newdata, type="probabilities")
SVManova$prob <- svmProb
svmSortanova <- function(x) x[order(x$C), ]
SVManova$sort <- svmSortanova

load data:

dataA2<-read.csv("C:/results/A2.txt",header = TRUE,
                             blank.lines.skip = TRUE,sep = ",")
set.seed(1)
inTrainSet <- createDataPartition(dataA2$R, p = 0.75, list = FALSE) #[[1]]
trainSet <- dataA2[inTrainSet,]
testSet <- dataA2[-inTrainSet,]
#-----------------------------------------------------------------------------
#K-folds resampling method for fitting svr
ctrl <- trainControl(method = "repeatedcv", number = 10, repeats = 10,
                     allowParallel = TRUE) #10 separate 10-fold cross-validations

link to data:

wuala.com/jpcgandre/Documents/Data%20SVR/?key=BOD9NTINzRHG


texts: what happens when a decision tree can't be split into further unit values?

texts:
Suppose I have a dataset
A B C D
1 1 1 0
1 1 0 0
1 1 0 1
1 0 1 1

Here A,B,C,D are my independent features and D is my dependent feature. Now if I make a decision tree here.
A -> Yes/No = 2/2 - > Based on value 1-> 2/2 So we go further for B.
B -> Yes/No = 2/2 -> Based on value 1-> 1/2 So we go further for C
When B=1 & C = 1 then D = 0
When B=1 & C = 0 then D = 0 and as well as D = 1 too.

So here we have a 50/50 chance, Or we can see we can't go further to get unique value to predict. So how does decision tree solve this problem?


texts:
I'm trying to train a Lane Detection CNN called PINet on a proprietary dataset.
Below are some of the important configuration values:

Batch size: 6
Optimizer: Adam
Learning rate: High of 1e-4 and Low of 1e-6
Lr scheduler: Cyclic lr scheduler in triangular mode
Dropout: 0.5
weight_decay: 4e-3

Below is the summary of my dataset:

The dataset contains 15200 samples in the train dataset and 2700
samples in the dev/test dataset.
For train dataset: 7000/15200 samples come from a stationary scene.
I.e. 7000 samples display almost exact same scene.

Below you can find the learning curves I have plotted during training
My questions are:

As per my analysis, The model has achieved an overfit. Am I right
here?? Looking at the Train/Validation loss what would you say about
the model fit type??
Why my training and validation losses are almost stable (with small
variation)??
What can be done to reduce the training loss even further??
What can be done to reduce the gap between Validation and training
loss?? is it advisable to use dropout values of 0.6 or 0.7 and weight decay as high as 1e-2 to reduce overfitting??


texts:
When I use data augmentation to increase the train dataset, should I use all augmentation techniques (parameters in keras)?
Which data augmentation parameters should use with flow_from_directory?


texts: Getting mean and covariance matrix for multivariate normal from keras model

texts:
I have a dataset that has 6 input features and 5 output features. I want to use a keras sequential model to estimate the mean vector and covariance matrix from any row of input features assuming the output features to be following Multivariate Normal Distribution.
That is for my dataset for any row of 6 input features, I want to get a mean vector of 5 values and a 5*5 covariance matrix.
sample=pd.DataFrame({'X1':[1,2,3,4,5,6],
              'X2':[1,3,1,5,2,7],
              'X3':[3,0,0,7,5,0],
              'X4':[0,4,3,2,5,8],
              'X5':[9,7,0,2,4,5],
              'X6':[1,1,8,7,0,0],
              'Y1':[0.5,1.2,6.3,4.5,1.5,6.6],
              'Y2':[6.1,4.3,2.1,1.5,4.2,8.7],
              'Y3':[0,0,3.2,3.7,5.5,0.2],
              'Y4':[0.5,1.4,8.3,5.2,1.5,1.8],
              'Y5':[2.9,1.7,6.3,5.2,9.4,1.5]})
sample

    X1  X2  X3  X4  X5  X6  Y1  Y2  Y3  Y4  Y5
0   1   1   3   0   9   1   0.5 6.1 0.0 0.5 2.9
1   2   3   0   4   7   1   1.2 4.3 0.0 1.4 1.7
2   3   1   0   3   0   8   6.3 2.1 3.2 8.3 6.3
3   4   5   7   2   2   7   4.5 1.5 3.7 5.2 5.2
4   5   2   5   5   4   0   1.5 4.2 5.5 1.5 9.4
5   6   7   0   8   5   0   6.6 8.7 0.2 1.8 1.5

For loss function I am using the following, which maximizes the log probability.
def lossF(y_true, mu, cov):

  dist = tfp.distributions.MultivariateNormalTriL(loc=mu, scale_tril=tf.linalg.cholesky(cov))
  return tf.reduce_mean(-dist.log_prob(y_true))

I am trying something like below, but getting confused in the middle.
#X_train has 6 values in each row
#y_train has 5 values in each row
#y_pred should be either a distribution function or mu & cov for each row

opt = Adam(learning_rate=0.001)
inputs = Input(shape=(6,))
layer1 = Dense(24, activation='relu')(inputs)
layer2 = Dense(12, activation='relu')(layer1)
predictions = ???
model = Model(inputs=???, outputs=???)
model.compile(optimizer=opt, loss=loss_fn)
model.fit(X_train, y_train, epochs=100, batch_size=100)
y_pred=model.predict(X_test)

Note: instead of getting mu and cov separately, if its possible to get distribution function as output that would be helpful too.


texts:
I'm pursuing a computer science minor at my university, and one class I'm in is Machine Learning.
We have a project to utilize a few algorithms we have learned so far.
I've been using SciKit-Learn to perform these algorithms, but when it comes to decision trees I keep getting a feeling I am overfitting.

I'm using a dataset about weather, giving characteristics such as city, state, month, year, wind direction, wind speed, etc... where the target variable is the average temperature for the day. Now I know this is hard to classify, as it is pretty much a continuous variable space, but I've simplified it to the predicted being within a range of 5 to the actual.

Here is a link to the csv I'm using

The following is my code:

address2 = 'C:/.../weather.csv'
weather = pd.read_csv(address2)

cityCode= le.fit_transform(weather.iloc[:,2])
windDirection = le.fit_transform(weather.iloc[:,3])
month = le.fit_transform(weather.iloc[:,8])
precip = le.fit_transform(weather.iloc[:,9])
windSpeed = le.fit_transform(weather.iloc[:,10])
state = le.fit_transform(weather.iloc[:,11])
week = le.fit_transform(weather.iloc[:,12])
year = le.fit_transform(weather.iloc[:,13])

Xweather = list(zip(cityCode,windDirection,month,precip,windSpeed,state,week,year))
yweather= weather.iloc[:,0]

yweather_test = train_test_split(Xweather, y, test_size = 0.2, random_state=413)

cWeather = tree.DecisionTreeClassifier()
cWeather.fit(Xweather_train,yweather_train)
accu_train_weather=np.sum(abs(cWeather.predict(Xweather_train)-yweather_train)<=5)/float(yweather_train.size)*100
accu_test_weather=np.sum(abs(cWeather.predict(Xweather_test)-yweather_test)<=5)/float(yweather_test.size)*100
print("Classificaton accuracy on training set", accu_train_weather, "%")
print("Classificaton accuracy on test set", accu_test_weather, "%")

My training set constantly gets 100% training accuracy, but the test set is constantly 57% accurate, which leads me to believe the tree is overfitting to the training set.

I know I'm not doing any pruning, but even when I do, I can get the same test accuracy as unpruned at best.
By pruning I mean setting the tree classifier to have a maximum number of leaves, minimum samples per leaf, and maximum depth.

I'm not an advanced coder (as you can probably tell by my code), but any help would be great.


texts: Please suggest some simple functions that I can use to test my neural network implementation

texts:
I am a beginner and I have developed code in Octave to train a neural network (as part of Andrew Ng's coursera course).

Now I would like to generate some datasets in Excel for simple mathematical functions and try and train my neural network implementation.

I tried y = x^2 and y = sin(x), but my training accuracy is only around 67%.

I have varied the number of hidden layer units, varied the lambda, but still don't get good results. Where could I be wrong?


texts: Is it true more CPU core is better for deep learning?

texts:
I just started to learn the deep learning in my free time. I was hoping to buy a laptop where I want to implement some small(alexnet) to medium(GoogleNet) networks maybe something bigger. I searched for the GPU. Everyone is suggesting minimum RTX 2060. However, it's vague about the CPU. I heard people saying that it's better to have more number of core in CPU. All I understand CPU only do the data pre-processing. What I want to know is it feasible to have a 4 core CPU(e.g. i5 9300H) with RTX 2060 having a tight budget? Will this configuration serve my purpose?
Another things is new 10th Gen Intel Core i7-10750H processor with up to 5.0 GHz3 have a 6 cores. While AMD Ryzen 7 4800HS have 8 cores. Which will be best for deep learning. Is the number of cores really matters?


texts:
Note: I've read How do subsequent convolution layers work? a few times, but it's still difficult to understand because of the parameters $k_1$, $k_2$, and many proposals (1, 2.1, 2.2) in the question. This seems to be complex for other people too, I think I'm not the only one (see a few comments like "I have just struggled with this same question for a few hours"). So here it is formulated with a particular specific example with no parameters, to grasp the idea more easily.

Let's say we have a CNN with:

input: 28x28x1 grayscale images (28x28 pixels, 1 channel)
1st convolutional layer with kernel size 3x3, and 32 features
2nd convolutional layer with kernel size 3x3, and 64 features

Keras implementation:

model = Sequential()
model.add(Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(Conv2D(64, (3, 3), activation='relu'))

Question: how does the 2nd layer work?

More precisely:

for the 1st layer:

input size: (1, 28, 28, 1)
weights size: (3, 3, 1, 32)  (good to know: the number of weights doesn't depend on input pixel size)
output size: (1, 26, 26, 32)

for the 2nd layer:

input size: (1, 26, 26, 32)
weights size: (3, 3, 1 32, 64)
output size: (1, 24, 24, 64)

How is the latter possible? It seemed to me that, in the 2nd layer, every input 26x26 image will be convolved with the 3x3 kernels of each 64 feature maps, but this could done for all the 32 channels!

Thus I had the feeling the output of 2nd layer should be (1, 24, 24, 32*64)

How does it work here?


texts: I know this was asked before, but I didn't get a definitive answer. I am trying to download a simple 17GB .txt file that contains the entire IPv4 address space. There must be a file like this already on the web.


texts:
Have anyone used Shark as repository from resulting datasets from Apache Spark?

I'm starting some tests with Spark and read about this database tecnology. Have anyone been using it?


texts: In Naive Bayes classifier how is P(sneezing,builder|flu) = P(sneezing|flu)P(builder|flu)?

texts:
Please refer to this literature:

According to Naive Bayes classification algorithm:

$P(sneezing,builder|flu) = P(sneezing|flu)P(builder|flu) $

where sneezing and builder are independent events.

How do they arrive at the above conclusion mathematically?

Is it something like:

$P(sneezing,builder|flu)$
$=P(sneezing \cap builder| flu)$
$= \frac{P((sneezing \cap builder) \cap flu)}{P(flu)}$


texts: What are the 2 classes of categories to help define a problem

texts:
While studying about machine learning, I've learnt the importance of defining your problem before getting started trying to model it.

I can see 2 types of problem categorification:

Supervised / unsupervised / reinforcement algorithms
Classification / clustering / regression / ranking

Example definitions found on the net:

First type:

Supervised Algorithms: The training data set has inputs as well as the desired output. During the training session, the model will adjust its variables to map inputs to the corresponding output.
Unsupervised Algorithms: In this category, there is not a target outcome. The algorithms will cluster the data set for different groups.
Reinforcement Algorithms: These algorithms are trained on taking decisions. Therefore based on those decisions, the algorithm will train itself based on the success/error of output. Eventually by experience algorithm will able to give good predictions.

Second type:

Classification: You want an algorithm to answer binary yes-or-no questions (cats or dogs, good or bad, sheep or goats, you get the idea) or you want to make a multiclass classification (grass, trees, or bushes; cats, dogs, or birds etc.) You also need the right answers labeled, so an algorithm can learn from them.
Clustering: You want an algorithm to find the rules of classification and the number of classes. The main difference from classification tasks is that you don’t actually know what the groups and the principles of their division are. For instance, this usually happens when you need to segment your customers and tailor a specific approach to each segment depending on its qualities.
Regression: You want an algorithm to yield some numeric value. For example, if you spend too much time coming up with the right price for your product since it depends on many factors, regression algorithms can aid in estimating this value.
Ranking: Some machine learning algorithms just rank objects by a number of features. Ranking is actively used to recommend movies in video streaming services or show the products that a customer might purchase with a high probability based on his or her previous search and purchase activities.

Do each type of categories have a name ?
And are these types correlated or independent ?


texts: Neural network with multiple correlated responses and separate predictors for each response

texts:
If you have a large number of correlated response variables, each with its own set of predictors, but you think the relationship of the response to the predictors is the same for all predictors, what neural network architecture should be used? This differs from the case where all responses have the same inputs.

For example, if for 1000 stocks over 5 years you predict monthly returns each month using the 12-month return of each stock and the price/earnings ratio of each stock,
you have 5*12 = 60 observations. Each observation has 1000 responses with 2 predictors for each response. Stock returns are positively correlated to each other, so the responses cannot be assumed independent.


texts: Train on multi-domains, then fine-tune on specific domain

texts:
Would it make sense to first train a model on images from multiple domains, and then do "fine-tuning" on one specific domain to improve its performance on it?

For instance, one could train an object detector based on cars camera recorded in NYC, Paris and Beijing, then continue training on Paris only.
For a model that would be deployed on Paris only, should we favor diversity or specificity? And does this training method has a name?


texts: is it bad to have many different measurements for the same target variable?

texts:
I'm working on a dataset that has repeated measurements for the same target variable.

When I don't change anything and create model, cross validation overfits with 0.99 score but in testset it gives around 0.39.

When I use mean, std, skew, quartiles for each measurement to have only one measurement for each feature, it gives a much better score.

Can anyone explain to me why? and when it is good to use the second method?

the original dataset looks like this (all numbers are fake):

id /measurement1/measurement2/.../target/
0-1/0.18283     /0.12855     /.../  1   /
0-2/0.1141      /0.38484     /.../  1   /
0-3/0.4475      /0.18374     /.../  1   /

and transformed dataset looks like this:

id /meas1_avg/meas1_std/meas1_skew/meas2_avg/meas2_std/.../target/
0  /0.28747  /0.183848/ 0.198384  /0.18484  /0.28474  /.../  1   /


texts: differentiating vector with different operation on each elements

texts:
I have some idea how backpropagation would work for loss function like:
loss=summation(predicted-true)^2
Where predicted and true are vector of same length and same operation throughout the elements.
Now in object localization problem my neural network's output vector's 0th element would denote probability of particular object being in image and rest 4 would tell about bounding box. Now my loss function would be roughly something like this:
loss=CategoricalCrossentropy(pred[0],true[0])+MSE(pred[1:4]-true[1:4])
This loss function worked fine in tensorflow and my NN localized the object as I expected.
My problem is that I am not able to understand  how mathematically differentiation would work when different operations are applied on different elements.


texts: Why does PAC learning focus on learnability of the hypothesis class and not the target function?

texts:
The definition of PAC learning is roughly

An algorithm is a PAC learning algorithm if it given enough data, for any target function, it asymptotically does as well as it possibly could given the functions it's capable of representing.

This definition seems sort of unambitious. In reality, I care more about approximating the target function well in an absolute sense, not just approximating it as well as my hypothesis class can muster. By some kind of no-free-lunch principle, it's probably not possible to be able to do this for all target functions, so we need to be satisfied with algorithms that learn well on a class of target functions. If I were to take a shot at defining a notion of "learning", it would look like this:

An algorithm learns a class $\mathcal V$ of random variables in $\mathcal X\times\mathbb R$ iff for any $(X, Y)\in\mathcal V$, then for any $\delta, \epsilon$, given enough i.i.d. samples of $(X, Y)$, the algorithm returns a function $h$ such that $P(|h(X)-Y|\leq\epsilon)>1-\delta$.
A class $\mathcal V$ of random variables is learnable iff there exists an algorithm that learns it.

I have two questions:

Why the emphasis on getting good bounds for approximating the hypothesis class, rather than the target function/distribution, when surely it's the latter we care about?
Is something like my definition also studied?


texts: Predicting electricity consumption in a household

texts:
Please I need your help in brainstorming on a problem. I have a community, in this community, every household has a 'power device' that powers the household. Households need to buy power units from a power company, and then, the 'power device' is loaded with the units in order to power the household. For example, if a customer pays 1000, he will be given units worth 9.3 to use in powering his household. This is sample of the data I have -

Customer ID | Date     | Amount | Units
3030         2019/01/23   1000    9.3
3829         2019/04/01   500     4.3
3030         2019/01/31   1000    9.3

My challenge -

Given this dataset, I want to be able to predict the units remaining in the 'power device' at any point in time.
For example, looking at the dataset above, how many units will be remaining as at 2019/04/04 for customer ID - 3829?

Is this dataset sufficient and if yes, what ML algorithm would you suggest?


texts: What does discriminator of a GAN should do?

texts:
A Generative Adversarial Network (GAN) consists of two sub-networks: (1) generator and (2) discriminator.

What does a discriminator should be able to do? Or more specifically, should it be able to distinguish (classify) a real object (for example a vector) from a generated one or should it be able to distinguish a set of generated vectors from a set of real vectors?

I tend to think that the second option is correct. However, if it is the case, how do we build a neural network that classifies a set of vectors instead of a vector?


texts: Which model can solve the "sequence demand" problem?


texts: Using Decision Tree methodology to identify Independent Variables for Multiple Regression

texts:
Have access to a dataset with hundreds of variables and millions of cases (American Community Survey).

Need to identify a small, manageable set of Independent Variables (IVs) to use for Multiple Regression.

One way to do this, of course, would be to use applicable theories to identify the IVs.

Was wondering how I could use a data-driven (data-mining?) approach as follows:

Use a Decision Tree to identify impactful (candidate? relevant?) IVs?
And then use these as the IVs in the Multiple Regression?

(Seem to remember reading once, in passing, that this approach to variable reduction is permitted.)

Tried searching on Google for articles that clarify the above, but the search terms are such that I keep getting hits to articles that compare Decision Trees and Multiple Regression.

So, if you know of articles and research papers that describe how to do the above, please leave links below. Also, I would welcome your own original suggestions on how to proceed.


texts: Chat Bot Answering based on Data Corpus Self-Training

texts:
I have created a very simple chat bot based on RASA NLU. In this case, I manually create some sample input text and create a model for using it against unknown source of input. It's fine for now.

As my next step of learning, I want to do use a big documnet as source for my chat bot.

What steps should I do to make the program train automatically on the documnet text corpus and able to answer based on the user query? I want to avoid the manual training.

For this NLU problem statement, what libraries can I use?

Almost all internet sources talk about SQuaD models. How they are related to a custom domain training?

Any blogs, tutorials, libs that can help me to do this will be useful.

Some other related questions but without solid answers:

Build knowledge bot using deep learning


texts:
Need help with identifying a representation sample size 'n'. Let's say I have a very large population- infinite number of participants. I am picking the random sample from this infinite population. I want to get the most accurate results about my data set with involving the least amount of people; Pick 'n' that would be as small as possible. What 'n' / amount of people I need to pick from the sample to get the most accurate results with the least amount of people possible and how? I am really confused and do not know how to start. Any information would be appreciated.
What I have done so far: Since the population is infinite, I looked at it as at the distribution. So the smaller segment of the population would have the same distribution as the whole population. But how do I figure out the most accurate size of my random sample if I have infinite amount of population? I also applied the formula for the sample size: SS = [Z^2p (1 − p)]/ C2
where C is the confidence level and p is the percentage of population. But how do I know the percentage of population if it is infinite?


texts: Using predicted probabilities and bayesian inference to update beliefs

texts: I'm currently working on a project to predict the likelihood of an outcome. I'd like to implement a system where the belief the event will happen is updated after running the model on new data. However, I'm having trouble figuring out exactly how to implement that. Any explanations of updating beliefs using Bayes theorem would be helpful.


texts: What are the current killing machine learning methods?

texts: I was wondering whether we could list machine learning winning methods to apply in many fields of interest: NLP, image, vision, medical, deep package inspection, etc. I mean, if someone will get started a new ML project, what are the ML methods that cannot be forgotten?


texts:
I am trying to run Tensorflow for image recognition (classification) in Java (JSE not Android).

I am using the code from here, and here.

It works for Inceptionv3 models, and for models retrained from Inceptionv3.

But for MobileNet models it does not work, (such as following this article).

The code works, but gives the wrong results (wrong classify labels). What code/settings are required to use MobileNet from Java?

The code that works for Inceptionv3 is

try (Tensor image = Tensor.create(imageBytes)) {
            float[] labelProbabilities = executeInceptionGraph(graphDef, image);
            int bestLabelIdx = maxIndex(labelProbabilities);
            result.setText("");
            result.setText(String.format(
                            "BEST MATCH: %s (%.2f%% likely)",
                            labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));
            System.out.println(
                    String.format(
                            "BEST MATCH: %s (%.2f%% likely)",
                            labels.get(bestLabelIdx), labelProbabilities[bestLabelIdx] * 100f));
        }

This works of Inceptionv3 models, but not MobileNet,
Gives the error, "Expects args[0] to be float but string is provided"

For MobileNet we tried the code,

try (Graph g = new Graph()) {
  GraphBuilder b = new GraphBuilder(g);
  // Some constants specific to the pre-trained model at:
  // https://storage.googleapis.com/download.tensorflow.org/models/inception5h.zip
  //
  // - The model was trained with images scaled to 224x224 pixels.
  // - The colors, represented as R, G, B in 1-byte each were converted to
  //   float using (value - Mean)/Scale.
  final int H = 224;
  final int W = 224;
  final float mean = 128f;
  final float scale = 1f;

  // Since the graph is being constructed once per execution here, we can use a constant for the
  // input image. If the graph were to be re-used for multiple input images, a placeholder would
  // have been more appropriate.
  final Output<String> input = b.constant("input", imageBytes);
  final Output<Float> output =
      b.div(
          b.sub(
              b.resizeBilinear(
                  b.expandDims(
                      b.cast(b.decodeJpeg(input, 3), Float.class),
                      b.constant("make_batch", 0)),
                  b.constant("size", new int[] {H, W})),
              b.constant("mean", mean)),
          b.constant("scale", scale));
  try (Session s = new Session(g)) {
    return s.runner().fetch(output.op().name()).run().get(0).expect(Float.class);
  }
}

This works, but gives the wrong labels.


texts: Is 25-75 a good case-control ratio for a neural network?

texts:
I have 50,000 DNA samples from a biobank. Only 2,200 samples have the label (disease) that I want to work with for supervised learning in a neural network.

So my question is, how many of those non-diseased samples do I need to include in my training data? If the diseased samples are my cases, how big should my control group be? Twice as big? The same size? What would a good ratio be?

The feature space is insanely big, so I am worried about training times. It feels unnecessary to include all the other 48,000 controls.

And yes, yes I know it will depend on the accuracy that I am seeing in the model, but what’s a good starting ratio?


texts: Sampling for the encoder part of the VAE

texts:
my question regards the code utilized to implement the sampling function in the encoder part of VAE.
Supposing that we chose a latent dimension of 2. Before the latent representation, we have 4 neurons (2 for mu and 2 for sigma).
We arrive to this representation with a series of Conv2D on a MNIST dataset. Then for having the latent space, from theory, I know that we have to use the reparametrization trick to sample in a correct way.
So I have a function like this that has to do the job.
def sampling_z(args):
  mu, sigma = args
  batch     = K.shape(mu)[0]
  dim       = K.int_shape(mu)[1]
  eps       = K.random_normal(shape=(batch, dim))
  return mu + K.exp(sigma / 2) * eps

From this code in Python it seems that the information for the calculation of the normal needed for the trick are two values that I can extract from the two elements that compose the mu array.
Am I correct on supposing this? If yes why the shape of these two neurons (mu) contains information for the normal distribution? Probably there is something that I cannot understand on the following python functions:
K.shape,  K.int_shape

Why It is so important having the shape of a 2 dim array? As you can see I am little bit confused about this. Could please someone help me with this? Maybe is the mu that is not a 2d array.
Many thanks:-)


texts: Transforming words in sentences into vector form to prepare a model

texts:
I want to build a simple classifier that classifies if the text is a question or just a simple message. I understand logistic regression and can work to create a simple neural network.

I have the labeled input data in English, Japanese, Korean, Thai. How could I transform this data before I feed it into the classifier?


texts: Question about removal of duplicates in NLP, when specifically working on twitter data

texts: Should we remove the duplicates before pre-processing or after pre-processing in NLP specifically when working on the twitter data? Actually main concern for me is that duplicates after pre-processing may have come from different tweets and so removing duplicates after pre-processing would be incorrect right? These are my thoughts, can confirm and give a clear insight about this?


texts:
Will saving a trained model this way give me a model trained on every chunk of data or just the last chunk?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text = chunk['body']
  label = chunk['user_id']
  print(text.shape, label.shape)

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3)
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

For example, if the first chunk had labels 1 and 2, and the second chunk 3 and 4, will the final model be able to predict just 3 and 4? Or 1 and 2 as well, given the testing data has all the labels. Any help?

UPDATE
The chunk is used to get text from the csv. I have updated my code.


texts:
I was learning Classification-based collaboration system and while running the code I faced the error AttributeError: 'DataFrame' object has no attribute 'ix'. Here is the code I write until now.
X=bank_full.ix[:,(18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36)].values


texts: What algorithm to use for extracting information from bank statements

texts:
I am trying to find the best way to extract information from bank statements. A bank transaction is not a natural text but still human readable.

I would like to extract a bunch of data if present like payment method, date, amount, vendor/customer name and even information like an order/invoice ID or the reason. I have sets of data that I can use for the training, for instance (100k+ vendors, payment methods words, etc). Also the solution must work with multi languages. The bank transactions could be in different languages (not only english).

Is named entity recognition the best way to go?

here is some sample data I can have in input:

Dividend
POS PURCHASE Non-PIN: RCI*RINGCENTRAL, INC.
IMP.DEB.LEY 83652 - ALICUOTA GENERA
DIRECTDEBIT EQUITY COMMERCE MERCH FEES 158713 62000329526385 CCD
ADVANCE AUTO PARTS JACKSONVILLE

Thanks for your help


texts: Can a novelty detection model overfit? In novelty detection, the model is trained on normal data instances (not polluted by outliers) where no labels are used in the training process, while validated and tested on a data instances that contain outliers in them. An example of algorithms that can be used for novelty detection are one-class SVM (OCSVM) and Local Outlier Factor (LOF).


texts: How can I get the original pixels that lead to the decision in CNN?

texts:
I work on medical images and I want to locate the most relevant regions of the image based on deep learning spatially CNN. So I feed my data into VGG16 architecture and get the features maps. Now I want to figure how can I go back from the feature maps in the original image to recover the positions of areas? How can it be done?
Most of what I found is related to a detection object, like RCNN, but that is not what I'm searching for. I need to recover all the areas that CNN pay attention to, not only that depends on one class.


texts:
I'm wondering if the following strategy has been already used and could work

Let's says you have a CNN which work well to classify image data, dog and cat. You only have cat and dog image as training data.
Is there any way to use it to detect image of horse as anomaly?

For example, with a ruled based system we could says

if P(cat) and P(dog) ~0.5 then it's an anomaly

another way could be to take feature vector at last fully connected layer and compare vector very different from other could be considerated as anomaly

Do you have any related paper? is it a totaly dumb idea?


texts: Where can I download a tagged dataset of text related to finance, programming, analytics etc.?

texts:
I want to create and train a model which classifies a new text content into finance, programming, analytics, design etc. Where can I get a relevant dataset to train my models?

TIA.


texts:
I have build the model to detect the distance. So i need image of camera to be sent to my server per 100 millisecond. But
If this is not cool with low internet speed.
So i use js tool and able to run model on website itself without sending it to server. Now everything is perfect but i think any body can copy logic from client side. So do we have any way of setting logic to client side ( website or apps ) ?


texts: How to choose the model parameters (RandomizedSearchCV, .GridSearchCV) or manually

texts:
Faced with the task of selecting parameters for the lightgbm model, the question accordingly arises, what is the best way to select them? I used the RandomizedSearchCV method, within 10 hours the parameters were selected, but there was no sense in it, the accuracy was the same as when manually entering the parameters at random. +/- the meaning of the parameters is clear, which ones are responsible for retraining, which ones are for the accuracy and speed of training, but it’s not entirely clear if you select manually one at a time or in pairs, or even more options?

Below is an example of how I implemented the selection of parameters:

SEED = 4
NFOLDS = 2
kf = KFold(n_splits= NFOLDS, shuffle=False)

    parameters = {
          'num_leaves': np.arange(100,500,100),
          'min_child_weight': np.arange(0.01,1,0.01),
          'feature_fraction': np.arange(0.1,0.4,0.01),
          'bagging_fraction':np.arange(0.3,0.5,0.01),
          'min_data_in_leaf': np.arange(100,1500,10),
          'objective': ['binary'],
          'max_depth': [-1],
          'learning_rate':np.arange(0.001,0.02,0.001),
          "boosting_type": ['gbdt'],
          "bagging_seed": np.arange(10,42,5),
          "metric": ['auc'],
          "verbosity": [1],
          'reg_alpha': np.arange(0.3,1,0.2),
          'reg_lambda':  np.arange(0.37,0.39,0.001),
          'random_state': [425],
          'n_estimators': [500]}

model = lightgbm.LGBMClassifier()
RSCV = RandomizedSearchCV(model,parameters,scoring='roc_auc',cv=kf.split(train),n_iter=30,verbose=50)
RSCV.fit(train,label)


texts:
If I have fitted training data to a probability distribution, e.g. a poisson distribution, how can I test this fit on some test data?

To fit the poisson distribution I am using R's fitdistrplus package that using MLE for determining the optimal coefficients of a given distribution. Therefore, I have the estimated $\lambda$ for a poisson distribution based on my training data but I am not sure how to test this on some unseen test data.


texts: How many neural networks can a GPU run simultaneously?

texts: I am developing an application that would take feed from multiple IP cameras and run 3 different neural networks on it. I want to determine the number of such IP cameras that can I process parallelly on my GPU.


texts: Unable to interpret this 100% stacked area chart

texts:
I suffer from "graphical dyslexia". I have trouble interpreting charts. So I pore over them for long periods of time. For the life of me, I can't figure out how the conclusion from this chart is:

  the bottom 80% of men are fighting over the bottom 22% of women and the top 78% of women are fighting over the top 20% of men (Source)

Please help me understand.


texts: How to train a neural network model in python or any language that can train itself from a excel file and validate itself also from a excel file?

texts:
I have a large amount of data. I need to make a neural network that trains itself  from an excel data sheet and validates its output from another excel data set.
There will be an output coloumn in the validate excel file. The model needs to give the output near about the givern row.


texts:
I am using my company computer since I don't have another one or linux. Therefore, I am starting to use cloud resources to perform some tasks.

I have a very simple question: Since most cloud resources don't have a GUI, how can I perform simple checks e.g. visualizing the bounding boxes my algorithm has found on a picture?

How do people performing such tasks usually accomplish this? Is there an easy fix or do I have to go through the installation of GUI on the cloud instance? Or, do you usually just download the results locally and view them locally?


texts: How to order the data with respect to data type

texts:
I am having large data set (82 variables)
Is there any way to arrange data such a way that I have to get all numerical variables firstly then categorical variables so that I can run hypothesis testing or exploratory data analysis(eda) by having loop.

If not is there any way to do eda in simpler manner because I can't check the correlation or chi.square test for each variable


texts: Clustering Weekday Weekend Data and Multicollinearity

texts:
Hi I have data of weekday and weekend step counts in which I extracted metrics from them such as the wd steps, we steps, standard deviation of wd steps, standard deviation of we steps and so on...

  wd_count  we_count  wd_sd_count  we_sd_count  ... .... ....
1  5000      3000      300          500
2  7000      2000      400          100

If I do clustering on this data, the weekday and weekend variables are going to be highly correlated and I will have to remove them before clustering. Is there any way around this problem for this kind of analysis?


texts: Text detection on English and Chinese languages

texts:
https://arxiv.org/abs/1910.07954
In this paper, we have a convolutional character neural network where we have object detection by taking a character as a basic unit. First, we do character detection and recognition and then we go for text detection.

Here (Page number 5 under the subheading Iterative character detection) it is written that a model trained on English and Chinese texts will generalize well in regards to text detection.
But how English and Chinese texts are good for generalization in text detection?
If you have any queries regarding the paper you can ask me in the comment section
Thanks in advance!


texts: What is the top-5 error rate in ILSVRC challenge?

texts: While I'm reading the deep learning book, I came across something as "top-5  error rate". I want to know what is it really about and how it is evaluated in   ILSVRC (ImageNet Large Scale Visual Recognition challenge) ?